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Abstract 

A quantitative research synthesis (meta-analysis) was conducted on the literature concerning the 
effects of feedback on learning from computer-based instruction (CBI). Despite the widespread 
acceptance of feedback in computerized instruction, empirical support for particular types of 
feedback information has been inconsistent and contradictory. Effect size calculations from 22 
studies involving the administration of immediate achievement posttests resulted in a weighted 
mean effect size of .80. Also, a mean weighted effect size of .35 was obtained from 9 studies 
involving delayed posttest administration. Feedback effects on learning and retention were found 
to vary with CBI typology, format of unit content and access supplemental materials. Results 
indicate that the diagnostic and prescriptive management strategies of computer-based adaptive 
instructional systems provide the most effective feedback. The implementation of effective 
feedback in computerized instruction involves the computer's ability to verify the correctness of 
the learner's answer and the underlying causes of error. 
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The Effects of Computer-Presented Feedback on Learning from 
Computer-Based Instruction: A Meta- Analysis 

In general, the term feedback refers to evaluative information that is provided on the 
functioning of a system that is intended to correct variations from a productive pathway. It is 
widely recognized that performance feedback is an important ingredient in any effective learning 
environment (Glaser & Gooley, 1973; Lysakowski & Walberg, 1982), but this is especially so for 
computer-based learning conditions where human tutors are generally not built into the 
instructional system. 

Definitions and terminology in the area of instructional feedback abound, ranging from 
the now outdated S-R notion of feedback as reinforcement to more recent cognitive perspectives 
deriving from the information processing tradition. One common definition focuses on the 
informational aspect as student answers to instructional queries (Mory, 1992). Another more 
radical view (Schoderbeck, Schoderbeck & Kefalas, 1990) argues that feedback restores 
equilibrium to learning systems by providing the opportunity for the continuation of activities 
that are proceeding in a desirable and productive direction. 

The motivating aspects of feedback are often encountered in the literature. Lepper and 
Chabay (1^85) describe two categories of motivational feedback, commiseration and 
encouragement. In both cases motivational feedback attempts to influence the learner's behavior 
in a lesson by providing incentives for correct behavior or by issuing deterrents for undesirable 
behavior (Sales, 1988). 

It is generally agreed that feedback is a critical component of instruction. For instance, 
feedback in CBI can range from the very simple issuing of right-wrong statements (i.e., 
knowledge of results [KGR]) to more elaborate corrective statements (i.e., elaborate feedback). 
More recently, attempts have been made to provide adaptive feedback (e.g., Minnesota Adaptive 
Instructional System) that adjusts to the individualized learning needs of students. Clearly all of 
these configurations cannot be equally effective, and so it is the purpose of this paper to judge the 
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relative effectiveness of feedback in general and based on various CBI typologies. It will also 
address the issue of the effectiveness of immediate versus delayed feedback issued by computers. 

The technique that will be used to assess the effectiveness of computer-based feedback is 
called meta-analysis (Glass, McGaw & Smith, 1981). Essentially this approach to reviewing the 
literature in a particular area attempts to establish a standardized difference between a treatment 
condition (i.e., feedback) and a control condition (i.e ; , no feedback). This standardized metric is 
called an Effect Size and can be combined with the effect sizes from a variety of studies to 
produce an average indice of effectiveness for an instructional treatment. Effect sizes are often 
converted to percentile to improve their interpretability. For example, an effect size of 1.0 says 
that 84% of the treatment subjects performed higher than subjects at the mean of the control 
condition 

Cohen (1969) has provided rough guidelines for interpreting effect sizes. Small effects 
sizes (e.g., .20 or the 58th percentile) are similar to those associated with comparisons among the 
heights of 15 and 16 year old girls. Medium effect sizes (e.g., .50 or the 69th percentile) would 
be similar to differences between 14 and 18 year old girls. Large effect sizes (.80 or the 79th 
percentile) are of the order of magnitude of differences in IQ between holders of Ph.D. degrees 
and the average college freshman. 

In the general area of feedback, four meta-analyses have been accomplished. Bangert- 
Drowns, Kulik, Kulik and Morgan (1991) selected feedback studies involving the instructional 
effect of feedback in test-like situations. This resulted in an average effect size of .26. In another 
meta-analysis (Kulik & Kulik, 1988) investigated the effects of the timing of feedback in verbal 
learning, resulting in an estimated effect size of .34. Lysakowski and Walberg (1982) found the 
highest effect for corrective feedback (ES = .97) especially when it is combined with the 
instructional effects of cues, and student participation. 

Only one meta-analysis (Schimmel, 1983) has investigated the effects of feedback on 
learners in computerized and programmed instruction. This study resulted in a finding of .47 
standard deviations above control conditions. This is considered a medium effect size (Cohen, 
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1969). Since CBI and paper-based programmed instruction were mixed, a estimate of the pure 
effects of feedback in computerized instruction is still not available. 

Two methods for selecting studies for inclusion in meta-analyses have been articulated 
and roundly debated. Glass (1976) advocates the use of all studies in the area of interest, 
regardless of methodological flaws and other problematic issues. By contrast, Slavin (1986) has 
made the case for the extreme restriction of selection criteria. He argues that only 
methodologically sound studies should be averaged resulting in a "best evidence" synthesis of 
the research findings. The current study sought to establish a fair balance between these 
approaches in order not to inordinately restrict sample size while maintaining empirical integrity. 
The inclusion criteria for this study are presented in the next section. 

Method 

A quantitative meta-analysis of the literature on the effects of computer presented 
feedback on learning from computer-based instruction was conducted. Numerous meta-analytic 
procedures (Hedges & Olkin, 1985; Hunter & Schmid, 1990; Rosenthal, 1991) were used to 
synthesize empirical studies gathered during data collection. 

The overall quantitative meta-analytic approach involved the following steps: a) data 
collection; b) creation of inclusion criteria; c) development of a coding scheme and subsequent 
separate analysis and subsequent separate analysis of studies based involving immediate and 
delayed administration of feedback; d) calculation of effects sizes; e) calculation of unbiased 
estimators of effect sizes; f) calculation of average effect sizes; g) calculation of average 
weighted effect sizes; h) calculation of heterogeneity of effect sizes; i) calculation of frequencies 
and means for all categories across each study variable; j) calculation of one-way analyses of 
variance between all categories across each study feature, and k) performance of a multiple 
regression analysis on particular study features in studies involving immediate posttest 
administration. 
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Data Collection 

The data collection procedure involved acquiring all relevant empirical studies related to 
the effects of computer-based feedback on learning from computer-based instruction. The 
inclusion of ail possible studies, which met the, criteria, ensured that the meta-analysis comprised 
a representative sample of empirical studies on the topic. 

Multiple sources of information were searched to identify all relevant research studies. 
First, on-line searches were conducted probing various interdisciplinary resources. For example, 
the ERIC database was searched from 1966 to 1992 using various descriptors such as CAI, CAI 
and feedback, CAL, CAL and feedback, feedback, knowledge of results, review, programmed 
instruction, programmed instruction and feedback. The PsychLIT database was also searched 
from 1983 to 1992 using the CAI, CAI and feedback CAL, CAL andfeedbac'. feedback, feedback 
in CAI, computer-based instruction and feedback and learning. The Social Citation Index and 
MUSE (McGill University Libraries' Online Catalogue) were also probed using the keywords 
computer-assisted instruction (CAI), computer-assisted learning (CAL), feedback, knowledge of 
results, learning processes, computing research, programmed i.istruction, coaching and other 
possible combinations. 

Manual searches were conducted to gather research studies that were not catalogued as 
part of on-line databases and to identify current studies. The Educational Technology Abstracts, 
British Education Index, Dissertation Abstracts International, Masters Abstracts, Current Index 
to Journals in Education, American Education Research Association's annual meeting program 
handbooks, card catalogues of Concordia University and textbooks related to the topic were also 
searched using a wider set of keywords, including computing research, programmed instruction, 
coaching, learner model, confirmation, knowledge-based systems, artificial intelligent systems, 
advisement, knowledge of results, intelligent tutoring systems, and videodisc. 

Research studies which were inaccessible (e.g., master's theses, dissertations, conference 
papers) through existing library facilities, including Inter-Library Loans, were acquired by 
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contacting the primary author. Thirty-four (34) dissertations were identified as pertinent to the 
meta-analysis, but only four were received through mail contacts. 

The "ancestry approach" of identifying pertinent articles was conducted by inspecting the 
reference section of all the empirical studies and theoretical articles that were collected via on- 
line and manual searches. This examination prevented the omission of relevant articles that 
would have otherwise been excluded from the analysis. 

Inclusion Criteria 

Fifty-nine studies (59) were collected and critically evaluated for possible inclusion in the 
meta-analysis. The inclusion criteria were derived from the meta-analytic literature (e.g., Hossler 
& Scalese-Love, 1989), previous research findings on computer-based instruction (e.g., Clark, 
\9S3; 1984; 1985a; 1985b), previous meta-analytic reviews on feedback (e.g., Bangert-Drowns, 
Kulik, Kulik & Morgan, 1991) and other empirical features related to both the theoretical and 
statistical outcomes of the analysis (e.g., Bernard & Naidu, 1990). Acquired studies were 
evaluated systematically, and if they met the following criteria, were subsequently included in 

the quantitative meta-analysis: 

1 . Study compared an experimental group receiving computer-presented feedback with a 
control group receiving no computer-presented feedback (at a minimum one group 
each); 

2. Computer-presented feedback was administered following learner responses 
(avoided pre-search availability); 

3. Study provided operational definitions of all feedback and control conditions; 

4. Criterion test of learning was administered either immediately following or sometime 
after each subject interacted with the computerized instructional lesson; 

5. Study provided measures of central tendency, variability, and quantitative statistical 
results for all experimental and control group; 

6. All groups, experimental and control, consisted of equal sample sizes; 

7. Studies with large sample sizes were included in the meta-analysis, defined by 
Hedges (1982) as n e , n c as equal or greater to 10 subjects. 
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In all, 22 studies (see Appendix A) were retained in the analysis and 37 were rejected 
because they failed to meet one or more of the above mentioned criteria. Of those rejected, 26 
contained one methodological and the remaining 1 1 contained moie than one flaw. 

Coding Scheme 

Previous meta-analyses related to CBI (Kulik & Kulik, 1986; Kulik, Bangert & Williams, 
1983) and feedback (Lysakowski & Walberg, 1982), motivational and cognitive effects of 
feedback in computer-based instruction (Sales, 1988), Clark's (1983; 1884; 1985b) criticism of 
learning from media, confounding evidence in educational computing research (Clark, 1985a) 
and other coding schemes (e.g., Hossler & Scalese-Love, 1989) were used in drafting the initial 
coding scheme for the extraction of study features. 

The studies that met the inclusion criteria were reviewed and their features were coded 
using an initial coding schema. Subsequently, all unused study variables were deleted and study 
characteristics were recoded and verified using the revised scheme. 

The revised coding scheme comprised of the following sections: a) document 
characteristics, b) subject characteristics, c) instructional materials, d) criterion test, e) 
methodological features, and f) feedback characteristics. Overall, 83 individual study features 
were taken into account. Following the coding of study features, the studies were subdivided 
based on the posttest administration, immediate or delayed, and ensuing parallel quantitative 
analyses were conducted. 

Quantitative Analysis 

The quantitative analytic procedures involved two separate major analyses based on a) 
studies involving the administration of immediate posttest and, b) studies involving the 
administration of a delayed posttest. Parallel but separate calculations were performed 
throughout the quantitative component of the meta-analysis. The information gathered in each 
study, involving immediate and delayed posttest administration, was recorded for subsequent 
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meta-analytic procedures. Such information included main effects F-values related to the 
feedback group, r-test statistic values, total sample size, experimental and control group sample 
sizes, experimental and control group standard deviations, and experimental and control group 
means. 

Results 

Immediate Posttest Data 

Nineteen effect sizes were extracted from fourteen studies reporting F values related to a 
main effect for feedback and the total sample size (Anderson, Kulhavy, & Andre, 1971; Arnone, 
&Grabowski, 1992; Bumgarner, 1984; Chanond, 1988; Clariana, Ross, & Morrison, 1991; 
Gaynor, 1981; Gilman, 1969a; Gilman 1969b; Hines, & Seidman, 1988; Hodes, 1985; Schaffer 
& Hannafin, 1986; Roper, 1977; Tennyson, 1981; Tennyson & Buttery, 1980). The effect size 
calculation involved two steps: transformation of the feedback main effect F value into a r-test 
statistic (Hunter & Schmid, 1990, p. 268), followed by the transformation of the t-test statistic to 
an effect size, d, using the formula described by Hunter & Schmid (1990, p. 272). 

One study (Anderson, Kulhavy & Andre, 1972) reported an independent r-test statistic 
comparing the difference between a feedback and control group with the respective degrees of 
freedom. The effect size, d, was calculated using a formula described by Rosenthal (1991, p.18). 

Fourteen effect sizes were calculated from five studies that provided the experimental and 
control group means, standard deviations and sample sizes (Armour-Thomas, White & Boehm, 
1987; Elliot, 1986; Johansen & Tennyson, 1983; Schloss, Wisniewski, & Cartwright, 1988; 
Tennyson, 1980). The effect size, g, was calculated using the formula described by Hedges and 
01kin(1985, p. 78-79). 

The fourteen effect sizes just described were converted into unbiased estimators of effect 
size, d, using the formula provided Hedges and Olkin, 1985 (p. 81) This resulted in an unbiased 
effect size estimator. 
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Average mean effect size. T he mean effect size for all the studies involving the effects of 
computer-presented feedback on learning as measured by the administration of an immediate 
posttest was calculated by summing the thirty-four effect size values (unbiased estimators) and 
dividing the total by thirty-four. 

The methodology described by Rosenthal (1991, p. 78) was used to calculate the 
estimated variance of effect sizes and subsequently the weighted mean effect size. Finally, the 
weighted mean effect size was calculated by applying the formula provided by Rosenthal, 1991 
(p. 78). 

The unweighted effect size was .73 with a standard deviation of .57. This effect was 
based on a total sample size of 2201 subjects. The overall weighted effect size was .80 (p < .05). 
The magnitude and direction of the effect size indicates that achievement outcomes were greater 
for the feedback group than the no-feedback control group. The individual effect sizes for each 
study included in the analysis along with sample size and CBI typology are presented in Table 1. 



Insert Table 1 about here 



Homog e neity of Effect Size, A test of homogeneity of effect sizes was performed to 
determine if the calculated effect sizes could be considered a homogeneous set. This test was 
carried out using the procedure described by Rosenthal, 1991 (p.79). rhe calculated value of A 2 
(k = 33, N = 22) = 558.60, exceeded the critical value of 43.77 (at alpha = .05). Therefore, the 
thirty-four effect sizes were considered heterogeneous. 

AN OVA on study features. For further statistical analyses, twenty-two of eighty-three 
study features were selected, based on comparable frequencies in each category. One-way 
ANOVAs for each of the following study features were calculated: document source; academic 
level; computer-based education typology; subject matter content; use of an instructional design 
model; computer-based education material construction; type of instructional content; 
instructional item type; whether CBI materials were modified version of preexisting materials; 
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format of unit content; accessibility to supplemental instructional materials; administration of 
criterion test; test material construction; task domain; instructional item type of criterion posttest; 
familiarity of criterion items; timing of criterion; validation of test items performed by; sample 
selection; participation; and strategy used by feedback group. 

Of the twenty-two study characteristics tested, six resulted in significant differences 
across coded categories. These were: academic level (primary, secondary or college), F (2, 23) = 
5.3, p < .05; CBI typology (linear CAI, Minnesota Adaptive Instructional System, computer- 
driven interactive Video or branching CAI), F (3, 33) = 14.3, p < .05; instructional systems 
design used (yes or no), F (1, 33) = 14.2, p < .05; format or unit content (graphics, text or 
multiple format), F (2, 7) = 45.0, p< .05; and accessibility to supplementary instructional 
materials (yes or no), F (1.33) = 16.5, p < .05. 

Multiple Regression. A stepwise regression analysis was conducted on study features to 
determine the possible relationship between these features and effect size. The search for the 
three study features with the lowest probability values is related to the inclusion of a single 
variable for each 10 subjects in an experiment (Tabachnick & Fidell, 1989). In this case, the 
total number of subjects or effect sizes was thirty-four (n = 34) and thus only three study features 
coulu be incorporated into the multiple regression analysis. Three study features with the lowest 
probability values, computer-based education typology, format of unit content, and accessibility 
to supplemental materials, were chosen for inclusion in the analysis. 

Delayed Po sttest Data 

The calculation of effect sizes for the delayed administration of feedback followed the 
same pattern as that just described for the administration of immediate. Five effect sizes were 
extracted form three studies (Chanond, 1988; Clariana, Ross & Morrison, 1991; Gaynor, 1981) 
reporting F values related to a main effect for feedback. 
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One study (Anderson, Xulhavy & Andre, 197 *) reported an independent r-test statistic 
comparing the difference between a feedback and control group with the respective degrees of 
freedom. An effect size was calculated using a formula described by Rosenthal (1991, p.18). 

Three effect sizes were calculated from one soidy (Armour-Thomas, White & Boehm, 
1987) that provided the experimental and control group means, standard deviations and sample 
sizes. The effect size, g, was calculated. 

Unbiased Estimators o f Effect Size. The effect sizes extracted from one study using the 
Hedges & Olkin (1985) methodology resulted in the calculation of a biased effect sizes, g, which 
were converted into an unbiased estimator of effect size, d. 

Average mean effect size. T he mean effect size for all th6 studies involving the effects of 
computer-presented feedback on learning as measured by the administration of a delayed posttest 
was calculated by summing the nine effect size (unbiased estimator) values and dividing the total 
by sample size. 

Weigh t ed Mean Effect Size. The methodology described by Rosenthal (1991) was used to 
calculate the estimated variance of effect sizes and subsequently the weighted mean effect size. 
Effect sizes, all positive, ranged from a low of .15 standard deviations to a high of .62. The 
unweighted mean effect size for the sample of 665 subjects was .34 with a standard deviation of 
.17. The overall weighted effect size was .35 (p < .05). The small effect size and its positive 
direction indicates that achievement outcomes were greater for feedback than non-feedback 
groups. This is the equivalent of raising achievement scores from the 50th percentile to the 64th 
percentile. The individual effect sizes for each study conducted, along with sample size and CBI 
typology are shown in Table 2. 



to test the homogeneity of effect size within the sample. The calculated value, X 2 (8, N = 9) = 



Insert Table 2 about here 




° iPi of effect size. The test of homogeneity of effect sizes was performed 
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25.39, exceeded the critical value of 15.51 (at alpha = .05) and therefore the nine effect sizes 
vere significantly heterogeneous. 

ANOVA procedure of study features, The frequencies and means of seventeen of eighty- 
three study variables were calculated. The study features included: document source, academic 
level, computer-based education typology, subject matter content, use of an instructional design 
model, type of instructional content, instructional item type, whether CBI materials were 
modified version of preexisting materials, whether subjects were introduced to the instructional 
unit, format of the unit content, accessibility to supplemental instructional materials, task 
domain, instructional item type of criterion posttest, timing of criterion, sample selection, 
participation, strategy used by feedback group. The calculation of one-way ANOVAs was 
inconceivable since the frequency distributions revealed study features with category sample 
sizes of one. 

Discussion 

Feedback has to be regarded as one of the most critical components of computer-based 
instruction, its objective being to provide students with appropriate responses thus allowing them 
to rectify learning impasses. In the best scenario, one-to-one human tutoring provides the 
potential for most flexible and individualized form of feedback. In contrast, computers must be 
programmed in advance to recognize learning difficulties and provide information that will aid 
the student in rectifying mistakes. Ideally, feedback messages should stimulate cognitive 
processes and strategies so that misconceptions will not be perpetuated, thus jeopardizing future 
learning attempts. 

The importance of feedback as a critical component of instruction and learning is 
exemplified by the magnitude and direction of the mean effect size involving studies with 
immediate posttest administration. The unweighted mean effect size of .80 indicates that 
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achievement outcomes were greater for the feedback group than the control group. This large 
effect size (Cohen, 1969) was interpreted as computer-presented feedback raising achievement 
scores by four fifths of a standard deviation. Equivalently, learners in the computer-presented 
feedback group performed better than 79% of the learners in the control group. 

The present study differs from meta-analytic studies that synthesized feedback-related 
issues involving immediate posttest administration. It was higher than meta-analyses involving 
the instructional effect of feedback in test-like situations (Bangert-Drowns, Kulik, Kulik & 
Morgan, 1991), the timing of feedback in verbal learning (Kulik & Kulik, 1988), but was smaller 
than Lysakowski and Walberg's (1982) quantitative synthesis of the instructional effects of cues, 
participation and corrective feedback. This may be due to the stricter inclusion criteria used in 
the present study. 

Compared to its Josest equivalent (Schimmel, 1983) the current study produced findings 
that are much more encouraging and suggestive of the potential of CBI feedback interventions. 
Schimmel's effect size of .47 may be have resulted from mixing apples and oranges (CBI and PI) 
or from the lack of incorporating studies involving adaptive feedback. 

The failure to meet the assumption of homogeneity of variance of effect size in the 
current study suggests that other intervening variables, such as type of computer-based typology 
may be a factor in the interpretation of the overall effect size obtained. Consistently, the 
approach referred to as the Minnesota Adaptive Instructional System (ES = 1.38) achieved the 
highest effect size, while Computer Driven Interactive Video (ES = 59), Linear CAI (ES = .49) 
and Branching CAI (.28) produced lower mean effect sizes. A test of these different computer 
feedback typologies produced one of the significant differences in follow-up analyses. 

The delayed posttest administration results indicated a decrease in long-term retention 
(ES = .35) as measured by achievement posttest scores compared to immediate posttest 
administration. A comparison of the general strategies used in the control and feedback groups 
of studies involving immediate and delayed posttest administration provides partial explanatory 
evidence as to the difference between the weighted mean effect sizes (.80 versus .35). 
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Studies involving immediate posttest administration compared control groups that 
presented the next screen of instruction following the learner's response without the presentation 
of a feedback message. In contrast, feedback groups generally included adaptive instructional 
approaches of continuous monitoring of student performance followed by the provision of advise 
concerning number of examples, and sequence of instruction as well as elaborative feedback 
messages. Conversely, studies involving delayed posttest administration incorporated control 
groups that omitted the presentation of a feedback message and proceeded to the next 
instructional sequence, while the feedback groups presented messages that differed in immediacy 
of delivery, format and content. It is clear from these results that immediate delivery of a 
feedback message provides the best instructional advantage to the student. 

This study has addressed an important and contradictory body of research literature using 
meta-analytic techniques in the area of computer presented feedback. Obviously, the number of 
studies excluded from consideration in this analysis bespeaks the somewhat methodologically 
weak state of research in the area. Naturally, qualitatively studies were omitted from this 
quantitative synthesis. The credibility and contribution of such works must be judged by other 
means, that are at this point in time still unrefined. However, it is encouraging to note that the 
sophistication that one would hope to observe in an emerging instructional technology seems to 
be evident in these results. As adaptive systems become more powerful, capable of offering 
tailor-made feedback based on dynamic assessment (e.g., Lajoie & Lesgold, 1989), effects due tc 
the inclusion of feedback in computer-based instruction should continue to raise achievement 
levels in computerized learning environments. 



Feedback in CBI 



16 



References 



Anderson, R. C, Kulhavy, R. W., 3c Andre, T. (1971). Feedback procedures in 
programmed instruction. Journal of Educational Psychology, 62(2), 148-156. 

Anderson, R. C, Kulhavy, R. W„ & Andre, T. (1972). Conditions under which 
feedback facilitates learning from programmed lessons. Journal of Educational Psychology, 
65(3), 186-188. 

Armour-Thomas, E., White, M. A., & Boehm, A. U987, April). The motivational 
effects of types of feedback on children's learning concepts and retention of relational 
concepts. Paper presented at the Annual Meeting of the American Educational Research 
Association, Washington, D.C. 

Arnone, M. P., & Grabowski, B. L. (1992). Effects on children's achievement and 
curiosity in learner control over an interactive video lesson. Educational Technology 
Research & Development, 40(1), 15-27. 

Bangert-Drowns, R. L., Kulik, J. A., & Kulik, C.L. C. (1984, August). The influence 
of study features on outcomes of educational research. Paper presented at the annual meeting 
of the American Psychological Association, Toronto. 

Bernard, R. M., & Naidu, S. (1990). Integrating research into instructional practice: 
The use and abuse of meta-analysis. Canadian Journal of Educational Communication, 
19(3), 171-198. 

Bumgarner, K. M, (1984). Effects of informational feedback and social reinforcement 
on elementary students' achievement during CM drill and practice on multiplication facts. 
Unpublished doctoral dissertation, Seattle University, WA. 

Chanond, K (1988, January). The effects of feedback, correctness of response and 
response confidence on learner's retention in computer-assisted instruction. Proceedings of 
selected research papers presented at the annual meetings of the Association for Educational 
Communications and Technology, New Orleans, LA. 

Clariana, R. B., Ross, S. M., & Morrison, G. R. (1991). The effects of different 
feedback strategies using computer-administered multiple-choice questions as instruction. 
Educational Technology Research & Development, 39(2), 5-17. 

Clark, R. E. (1983). Reconsidering research on learning from media. Review of 
Educational Research, 53(4), 445-459. 

Clark, R. E. (1984). Research on student thought process during computer-based 
instruction. Jo^nal of Instructional Development, 7(3), 2-5. 

Clark, R. E. (1985a). Confounding in educational computing research. Journal of 
Educational Computing Research, 1(2), 137-148. 

Clark, R. E. (1985b). Evidence for confounding in computer-based instruction 
studies: Analyzing the meta-analyses. Educational Communications Technology Journal, 
33(4), 249-262. 

Cohen, J. (1969). Statistical power analysis for the behavioral analysis. NY: 
Academic Press. 



ERIC 



IV 



Feedback in CBI 17 



Elliot, B. A. (1986). An investigation of the effects of computer feedback and interspersed 
questions on the text comprehension of poor readers. Unpublished doctoral thesis, Temple 
University, Phoenix. 

Gaynor, P. (1981). The effect of feedback delay on retention of computer-based 
mathematical material. Journal of Computer-Based Instruction, 8(2), 28-34. 

Gilman, D. A. (1969a). Comparison of several feedback methods for correcting errors by 
computer-assisted instruction. Journal of Educational Psychology, 60(6), 503-508. 

Gilman, D. A. (1969b). The effect of feedback on learner's certainty of response and 
attitude toward instruction in a computer-assisted instruction program for teaching science 
concepts. Journal of Research in Science Teaching, 6, 171-184. 

Glaser, R., & Colley, W. M.(1973). Instrumentation for teaching and instructional 
management. In R. W. Travers (Ed.), Second Handbook on Research on Teaching. Chicago, IL: 
Rand McNally. 

Glass, G. V. (1976). Primary, secondary, and meta-analysis of research. Educational 
Researcher, 5, 3-8. 

Glass, G. V., McGaw, B., & Smith, M. L. (1981). Meta-Analysis in Social Research. 
Beverly Hills, CA: Sage Publications. 

Hedges, L. V. (1982). Statistical Methodology in Meta-Analysis_(ERLC/TM Report 83). 
Princeton: University of Chicago, ERIC Clearinghouse in Tests, Measurements, & Evaluation 
Educational Testing Service. 

Hedges, L. V., & Olkin, I. (1985). Statistical Methods for Meta-Analysis. Orlando, FL: 
Academic Press. 

Hines, S. J., & Seidman, S. A. (1988, January). The effects of selected CAI design 
strategies on achievement, and an exploration of other related factors. Proceedings of selected 
research papers presented at the annual meeting of the Association for Educational 
Communications and Technology, New Orleans, LA. 

Hodes, C. L. (1985). Relative effectiveness of corrective and noncorrective feedback in 
computer-assisted instruction on learning and achievement. Journal of Educational Technology 
Systems, 75(4), 249-254. 

Hossler, D., & Scalese-Love, P. (1989). Grounded meta-analysis: A guide for research 
synthesis. The Review of Higher Education, 13(1), 1-28. 

Hunter, J. E., & Schmidt, F. L. (1990). Methods of meta-analysis: Correcting error bias 
in research findings. Newbury Park, CA: Sage Publications. 

Johansen, K. J., & Tennyson, R. D. (1983). Effect of adaptive advisement on perception 
in linear-controlled, computer-based instruction using a rule-learning task. Educational 
Communications and Technology Journal, 31(4), 226-236. 

Kulik, C-L. C, & Kulik, J. A. (1986). Effectiveness of computer-based education in 
colleges. AEDS Journal, 19(2), 81-108. 



18 



Feedback in CBI 18 



Kulik, J. A., Bangert, R., & Williams, G. (1983). Effects of computer-based teaching on 
secondary school students. Journal of Educational Psychology, 75, 19-26. 

Kulik, J. A., & Kulik, C-L., C. (1988). Timing of feedback and verbal learning. Review of 
Educational Research, 58( 1 ), 79-97. 

Lajoie, S., & Lesgold, A. (1989). Apprenticeship training in the workplace: Computer- 
coached practice environment as a new form of apprenticeship. Machine-Mediated Learning, 3, 
7-28. 

Lepper, M. R., & Chabay, R. W. (1985). Intrinsic motivation and instruction: Conflicting 
views on the role of motivational processes in computer-based education. Educational 
Psychologist, 20(4), 2 17-230. 

Lysakowski, R. S., & Walberg, H. J. (1982). Instructional effects of cues, participation, 
and corrective feedback: A quantitative synthesis. American Educational Research Journal, 
19(4), 559-57K. 

Mory, E. H. (1992). The use of informational feedback in instruction: Implications for 
future research. Educational Technology Research & Development, 40(3), 5-20. 

Roper, W. J. (1977). Feedback in computer assisted instruction. Programmed Learning 
and Educational Technology, 4(1), 44-49. 

Rosenthal, R. (1991). Meta-analytic procedures for social research (2nd ed.). Newbury 
Park, CA: Sage Publications. 

Sales, G. C. (1988). designing feedback for CBI: Matching feedback to the learner and 
learner outcomes. Computers in the Schools, 5(1-2), 225-239. 

Schaffer, L. C, & Hannafin, M. J. (1986). The effects of progressive interactivity on 
learning from interactive video. Educational Communications Technology Journal, 34(2), 89-96. 

Schloss, P. J., Wisniewski, L. A., & Cartwright, G. P. (1988). The differential effect of 
learner control and feedback in college students' performance on CAI modules. Journal of 
Educational Computing Research, 4(2), 141-150. 

Schoderbek, P. P., Schoderbek, C. G., & Kefalas, A. G. (1990). Management systems: 
Conceptual considerations. Boston: BPI, Irwin. 

Slavin, R. E. (1986). Best-evidence synthesis: An alternative to meta-analytic and 
traditional reviews. Educational Researcher, 15(9), 5-11. 

Tabachnick, B. G., & Fidell, L. S. (1989). Using multivariate statistics (2nd ed.). NY: 

Harper. 

Tennyson, R. D. (1980). Instructional control strategies and content structure as design 
variables in concept acquisition using computer-based instruction. Journal of Educational 
Psychology, 72(4), 525-532. 

Tennyson, R. D. (1981). Use of adaptive information for advisement in learning concepts 
and rules using computer-assisted instruction. American Educational Research Journal, 18(4), 
425-438. 



19 



Feedback in CBI 



Tennyson, R. D., Buttrey, T. (1980). Advisement and management strategies as design 
variables in computer-assisted instruction. Educational Communication Technology Journal , 
28(3), 169-176. 



20 



Feedback in CBI 20 



APPENDIX A 

Studies Included in the Meta-Analysis (n=22) 

Anderson, R. C, Kulhavy, R. W., & Andre, T. (1971). Feedback procedures in 
programmed instruction. Journal of Educational Psychology, 62(2), 148-156. 

Anderson, R. C, Kulhavy, R. W., & Andre, T. (1972). Conditions under which feedback 
facilitates learning from programmed lessons. Journal of Educational Psychology, 63(3), 186- 
188. 

Armour-Thomas, E., White, M. A., & Boehm, A. (1987). The motivational effects of 
types of feedback on children's learning concepts and retention of relational concepts. Paper 
presented at the Annual Meeting of the American Educational Research Association. 

Arnone, M. P., & Grabowski, B. L. (1992). Effects on children's achievement and 
curiosity in learner control over an interactive video lesson. Educational Technology Research 
& development, 40(1), 15-27. 

Bumgarner, K. M. (1984). Effects of informational feedback and social reinforcement on 
elementary students' achievement during CAI drill and practice on multiplication facts. 
Unpublished doctoral dissertation, Seattle University. 

Chanond, K. (1988, January). The effects of feedback, correctness of response and 
response confidence on learner's retention in computer-assisted instruction. Proceedings of 
selected research papers presented at the annual meetings of the Association for Educational 
Communications and Technology, New Orleans, LA, 

Clariana, R. B., Ross, S. M., & Morrison, G. R (1991). The effects of different feedback 
strategies using computer-administered multiple-choice questions as instruction. Educational 
Technology Research & Development, 39(2), 5-17. 

Elliot, B. A. (1986). An investigation of the effects of computer feedback and interspersed 
questions on the text comprehension of poor readers. Unpublished doctoral dissertation, Temple 
University. 

Gaynor, P. (1981). The effect of feedback delay on retention of computer-based 
mathematical material. Journal of Computer-Based Instruction, 8(2), 28-34. 

Gilman, D. A. (1969a). Comparison of several feedback methods for correcting errors by 
computer-assisted instruction. Journal of Educational Psychology, 60(6), 503-508. 

Gilman, D. A. (1969b). The effect of feedback on learner's certainty of response and 
attitude toward instruction in a computer-assisted instruction program for teaching science 
concepts. Journal of Research in Science Teaching, 6, 171-184. 

Hines, S. J., & Seidman, S. A. (1988, January). The effects of selected CAI design 
strategies on achievement, and an exploration of other related factors. Proceedings of selected 
research papers presented at the annual meeting of the Association for Educational 
Communications and Technology, New Orleans, LA. 



Feedback in CBI 21 



Hodes, C. L. (1985). Relative effectiveness of corrective and noncorrective feedback in 
computer-assisted instruction on learning and achievement:. Journal of Educational Technology 
Systems, 13(4), 249-254. 

Johansen, K. J., & Tennyson, R. D. (1983). Effect of adaptive advisement on perception 
in linear-controlled, computer-based instruction using a rule-learning task. Educational 
Communication and Technology Journal, 31(4), 226-236. 

Roper, W. J. (1977). Feedback in computer assisted instruction. Programmed Learning 
and Educational Technology, 4(1), 44-49. 

Schaffer, L. C, & Hannafin, M. J. (1986). The effects of progressive interactivity on 
learning from interactive video. Educational Communication and Technology Journal, 34(2), 89- 
96. 

Schloss, P. J., Wisniewski, L. A., & Cartwright, G. P. (1988). The differential effect of 
learner control and feedback in college students' performance on CAI modules. Journal of 
Educational Computing Research, 4(2), 141-150. 

Tennyson, R. D. (1980). Instructional control strategies and content structure as design 
variables in concept acquisition using computer-based instruction. Journal of Educational 
Psychology, 72(4), 525-532. 

Tennyson, R. D. (1981). Use of adaptive information for advisement in learning concepts 
and rules using computer-assisted instruction. American Educational Research Journal, 18(4), 
425-438. 

Tennyson, R. B., Buttrey, T. (1980). Advisement and management strategies as design 
variables in computer- assisted instruction. Educational Communication and Technology Journal, 
28(3), 169-176. 



22 



Feedback in CBI 23 



Table 2 



Effect Sizes Extracted from Studies Involving Delayed Posttest Administration 



/vuinors 


Lob i ypoiogy 


IN 


n e 


n c 


d 


Gay nor 1981 


Linear CAI 


92 






0.15 


Gaynor 1981 


Linear CAI 


92 






0.18 


Gay nor 1981 


Linear CAI 


92 






0.22 


Chanond 1988 


Linear CAI 


103 






0.26 


Armour- Thomas et al., 1987 


Linear CAI 


45 


22 


23 


0.28 


Armour-Thomas et al., 1987 


Linear CAI 


46 


23 


23 


0.30 


Armour-Thomas et al., 1987 


Linear CAI 


44 


21 


23 


0.42 


Clariana et al., 1991 


Branching CAI 


100 






0.60 


Anderson et al., 1971, Study 2 


Linear CAI 


50 


24 


26 1 


0.62 



Unweighted ES - .34 
Weighted ES = .35 
SD = .17 



N = total sample size 
ne = experimental group sample size 
nc = control group sample size 
d = effect size 



Feedback in CBI 22 



Table 1 



Effect Sizes Extracted from Studies Involving I mmediate Posttest Administration 



Authors 


CBl 




n c 




A 
U 




TvDOloIOBV 






Anderson etal, 1971, Study 1 


Linear CAI 


168 






0 03 
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41 


21 


20 


0 14 
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0 15 


Arnone et al., 1992 
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52 


25 


27 


0.18 


Gaynor 1981 
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92 






0.20 
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92 
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Schloss et al., 1988 


Linear CAI 


25 


7 


18 


0.31 


Gilman 1969b 


Linear CAI 
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0 38 
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25 


7 


18 


0 40 


Elliot 1986 
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42 


20 


11 


0.42 
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48 


24 


24 
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75 
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44 


21 


23 


0.54 
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25 


7 


18 


0.57 
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0.90 
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25 


7 


18 
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32 


16 
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63 
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23 
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32 


16 


16 
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32 


16 


16 
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Unweighted ES=.73 
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